Graph transformers are the state-of-the-art for learning from graph-structured data and are empirically known to avoid several pitfalls of message-passing architectures. However, there is limited theoretical analysis on why these models perform well in practice. In this work, we prove that attention-based architectures have structural benefits over graph convolutional networks in the context of node-level prediction tasks. Specifically, we study the neural network gaussian process limits of graph transformers (GAT, Graphormer, Specformer) with infinite width and infinite heads, and derive the node-level and edge-level kernels across the layers. Our results characterise how the node features and the graph structure propagate through the graph attention layers. As a specific example, we prove that graph transformers structurally preserve community information and maintain discriminative node representations even in deep layers, thereby preventing oversmoothing. We provide empirical evidence on synthetic and real-world graphs that validate our theoretical insights, such as integrating informative priors and positional encoding can improve performance of deep graph transformers.

Transformers may have sufficient supervision for generalization.

These limitations hinder GNNs' ability to capture long-range dependencies in graphs, ultimately

However, compared with the huge success of generative models in natural language processing [Tou-vron et al., 2023] and computer vision [Rombach et al., 2021], graph generation is faced with many

MPNNs, graph transformers do not rely on fixed graph structure information.

The formulation is very similar to the method for learning positional node embeddings. Asynthetic molecular graph regression dataset, where thepredictedscore isgivenby the subtraction of computationally estimated propertieslogP SA. Thetask is to classify the nodes into 2 communities, testing the GNNs ability to recognize predetermined subgraphs. For the training parameters, we employed an Adam optimizer with alearning rate decay strategy initializedin{10 3,10 4}asper[15],withsomeminormodifications: ZINC[15]. We selected aninitial learning rateof7 10 4 and increased thepatiencefrom 10 to 25 to ensure convergence.

The Transformer architecture has achieved remarkable success in a number of domains including natural language processing and computer vision.

Graphs arise across diverse domains, from biology and chemistry to social and information networks, as well as in transportation and logistics. Inference on graph-structured data requires methods that are permutation-invariant, scalable across varying sizes and sparsities, and capable of capturing complex long-range dependencies, making posterior estimation on graph parameters particularly challenging. Amortized Bayesian Inference (ABI) is a simulation-based framework that employs generative neural networks to enable fast, likelihood-free posterior inference. We adapt ABI to graph data to address these challenges to perform inference on node-, edge-, and graph-level parameters. Our approach couples permutation-invariant graph encoders with flexible neural posterior estimators in a two-module pipeline: a summary network maps attributed graphs to fixed-length representations, and an inference network approximates the posterior over parameters. In this setting, several neural architectures can serve as the summary network. In this work we evaluate multiple architectures and assess their performance on controlled synthetic settings and two real-world domains -- biology and logistics -- in terms of recovery and calibration.